Skip to content

fix(graph): fix stuck claims, cancels and artifact ids in Graph V1 - #1

Merged
SeasonsChange422 merged 3 commits into
mainfrom
fix/graph-v1-acceptance-fixes
Oct 14, 2026
Merged

SeasonsChange422 merged 3 commits into
mainfrom
fix/graph-v1-acceptance-fixes

Conversation

@SeasonsChange422

Copy link
Copy Markdown
Member

Summary

Three fixes for Graph (V1, serial) found while running it end to end against a real Codex runtime on a local build of main (ad25feb0c). Each has a regression test that fails on current main.

Commit Problem Effect before
fix(daemon): forward daemon instance id on WebSocket task claims The WS claim path rebuilt the HTTP request but dropped X-Daemon-Instance-ID, which Graph claims require Every Graph task claimed over WebSocket was rejected and requeued immediately in a loop; the node never started
fix(graph): carry physical stop evidence on cancelled, idle-killed and timed-out attempts executeAndDrain returned a bare cancelled result before the backend finished tearing down its process tree, so no stop receipt was sent A run cancelled mid-flight stayed in cancel_requested indefinitely
fix(graph): reject non-UUID artifact ids at submission and tell agents what to use ValidateResult only checked that an artifact id was non-empty, but ids are stored in a uuid column A produce_commit agent that used the commit SHA as the id was accepted, then reconciliation failed every 3 seconds forever and the run could not advance

Reproduction and verification

Local build of main at ad25feb0c, real Codex CLI (gpt-6-sol), database migrated to 357:

  • WS claim: 684 claim/requeue cycles in about a minute, agent never launched. With the fix, the same stuck task is claimed immediately and the node runs.
  • Cancel: cancelled while the node was executing, the run was still cancel_requested after about 85s. With the fix it reaches cancelled in about 9s.
  • Artifact id: the new unit test fails on main (artifact id "<sha>" accepted at submission). End to end on this build the agent happened to generate a UUID, so the wedge itself was observed on the Graph V2 branch rather than on main; the validation gap is identical in both.
  • A produce_commit run on main with the fixes completed, pushed refs/heads/multica/graphs/<run>/implement/1, and left the repository's main untouched.

Tests: the three new tests fail before and pass after. Across internal/graph, internal/daemon, internal/daemonws and internal/handler, the failing set on a Windows host is identical with and without the change (48 tests, all pre-existing).

Notes for release

  • Ship the server and the daemon/CLI together. The WS-claim and artifact validation fixes are server-side; the stop-evidence fix and the node prompt change are in the daemon. Updating only the server leaves the cancel problem in place.
  • A run that is already stuck in cancel_requested is not healed by this change; it needs a manual cancel or archive.
  • The executeAndDrain change only waits (up to 30s) when DisableWholeRunRetry is set, i.e. for Graph attempts. Ordinary tasks are not delayed; a test covers this.

Not verified

  • Whether production daemons claim over WebSocket, so I cannot say how many installs hit the first problem.
  • Linux CI and -race were not run (Windows host without CGO).
  • Only the Codex runtime was exercised; Claude on main was not tested.

Test plan

  • CI green on Linux, including go test -race ./server/internal/...
  • Start a Graph run on a daemon connected over WebSocket; the node starts without repeated claim/requeue
  • Cancel a run while its node is executing; the run reaches cancelled within seconds
  • Submit a git_commit artifact whose id is a commit SHA; the submission is rejected with a message asking for a UUID

🤖 Generated with Claude Code

The WS claim path rebuilt the HTTP claim request from the connection identity
but dropped X-Daemon-Instance-ID. Graph claims bind an attempt to the daemon
process identity and refuse a claim without it, so every Graph task a daemon
claimed over WebSocket was rejected and requeued immediately, in a loop, and
the node never started.

Reproduced on a local build of main (ad25feb) with a real Codex runtime: 684
claim/requeue cycles in about a minute, the agent never launched and the node
stayed queued. With this change the same stuck task is claimed at once and the
node runs.

Capture the header at connect, forward it when rebuilding the claim request,
and cover the forwarded headers with a regression test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d timed-out attempts

When the server cancels a task, executeAndDrain returns from its drainCtx.Done
branch immediately with a bare cancelled result, before the backend has finished
tearing down its process tree. The backend computes ExecutionStopped only after
that, so the result reached the daemon without stop evidence. A Graph attempt is
settled only from a stop receipt, so the daemon withheld it (execution_unknown)
and the engine, by design, kept the run blocked: a run cancelled mid-flight
stayed in cancel_requested indefinitely. The idle-watchdog and drain-timeout
branches had the same gap.

For Graph attempts (DisableWholeRunRetry), wait up to 30s for the backend's
terminal result in those branches and carry its ExecutionStopped. Ordinary
tasks never read the evidence and are not delayed.

Reproduced on a local build of main (ad25feb) with a real Codex runtime: a run
cancelled while its node was executing was still cancel_requested after 80s.
With this change it reaches cancelled in about 9s. A run that was already stuck
before the fix is not healed by it and needs a manual cancel/archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s what to use

A produce_commit agent declared its commit SHA as the git_commit artifact id.
The prompt never said what the id must be and ValidateResult only checked that
it was non-empty, so the submission was accepted. The id is stored in a uuid
column, so reconciliation then failed with "invalid_id: invalid Graph resource
UUID" every 3 seconds, permanently, and the run could not advance; only
cancelling it recovered.

Validate the artifact id as a UUID when the result is submitted, so the agent
gets a correctable error, and state the requirement in the node prompt and in
the built-in working-on-issues skill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@SeasonsChange422 SeasonsChange422 changed the title Fix/graph v1 acceptance fixes fix(graph): fix stuck claims, cancels and artifact ids in Graph V1 Oct 14, 2026
@SeasonsChange422
SeasonsChange422 merged commit 1c4ae01 into main Oct 14, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants